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ABSTRACT 

This paper describes the Global Real-time Assessment Teaching Tool for Teaching Enhance¬ 
ment (G-RATE). It is a tool framed around the elements of Bransford, Brown, and Cocking’s 
(1999) “How People Learn” framework and informed from data collected via laboratory obser¬ 
vations; focus group interviews with engineering supervisors and graduate teaching assistants; 
and undergraduate student surveys. The G-RATE allows various stakeholders (i.e., classroom 
observers, administrators, graduate teaching assistants, undergraduates, and students) to 
provide feedback about the pedagogical practices of graduate teaching assistants within a 
laboratory session. Future applications of the tool include the creation of pedagogical profiles 
for instructors and the dissemination of the tool across multiple educational environments. 

Keywords: Classroom assessment, engineering pedagogy, observation systems, graduate teaching 
assistants 
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INTRODUCTION 


Challenges facing graduate education in the United States include narrow disciplinary training of 
graduate students (COSEPUP, 1995; Nerad, 2004), declining enrollments of domestic students within 
doctoral engineering programs (COSEPUP, 1995; NRC, 2006; Akay, 2008), and an overwhelming 
absence of pedagogical training and feedback systems among engineering faculty and postdoctoral 
researchers (NSB, 2007). In fact, few engineering faculty obtain teaching experience as graduate 
students (Ruscio, 1987; Boice, 1991; McDermott, 1990; Reinarz, 1991; Shea and Taylor, 1990; Stice, 
Felder, Woods, and Rugarcia, 2000). Those who obtain teaching assistantships as graduate students 
often receive no formal pedagogical training (White, 1993; Bomotti, 1994; Cahn, 1994; Rushin et al., 
1997; Shannon, Twale, and Moore, 1998; Golde and Dore, 2001), and those who do obtain formal 
training often receive limited useful information about how to be effective graduate teaching as¬ 
sistants (GTAs) (White, 1993; Tang and Sandell, 2000). Those who participate in training via short 
and intensive programs, workshops, and courses before beginning of their roles as GTAs typically 
do not receive formative performance evaluations. Because of this, GTAs who later become faculty 
often rely upon lecture as their primary method of instruction within engineering courses (Rugarcia, 
Felder, Woods, and Stice, 2000; Cox, 2005). Numerous studies highlight the benefits of non-lecture 
pedagogical techniques which result in better student learning, persistence, and attitudes (Cooper 
and Robinson, 1998; Springer et al., 1998; Cudd and Wasser, 1999; Cabrera et al., 2001). 

The roles of engineering graduate teaching assistants vary from grading student assignments 
to teaching courses or laboratories independently depending on the content and context of the 
courses in which they are involved (e.g., traditional lecture or laboratory). Similarly, the formative 
and summative feedback needs of engineering GTAs may vary as well. The authors have developed 
the Global Real-time Assessment Tool for Teaching Enhancement (G-RATE) to provide feedback to 
GTAs for their improvement towards effective teaching and to understand the unique roles of engi¬ 
neering GTAs and their relationships with their supervisors (who are the ones most likely to provide 
them with pedagogical feedback). As an assessment tool that incorporates the pedagogical views of 
various stakeholders, the G-RATE addresses multiple perspectives by providing feedback from stu¬ 
dents, observers, administrators, researchers and GTAs. This paper presents the development of the 
G-RATE and a detailed description about the functionality of G-RATE. More specifically, it (1) reports 
the development of the G-RATE from observational data collected in engineering laboratory courses 
using a reliable and valid observational tool, the VaNTH Observation System (VOS) (Harris and Cox, 
2003; Cox and Cordray, 2008), (2) addresses what the VOS captures and what it does not capture in 
an engineering laboratory environment, (3) explores the different types of feedback that GTAs and 
supervisors need and the perceived roles of GTAs in engineering laboratory environments, and (4) 
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provides a detailed discussion and demonstration about the function and usability of G-RATE for the 
purpose of capturing pedagogical practices and providing feedback in engineering laboratories. 


LITERATURE REVIEW 


Graduate students as teachers and laboratory instructors play a big role in U.S. higher education 
and as such, spend a significant amount of time engaging with undergraduate engineering students. 
Since laboratory projects are a prerequisite for understanding engineering practice, it is imperative 
that individuals who teach or supervise in these laboratories possess the tools to improve their 
pedagogical practices. The definitions of a laboratory can differ in engineering (e.g., one-on-one 
troubleshooting sessions or senior design courses). 

The roles of GTAs (Allen and Rueter, 1990; Nyquist, Abbott, Wulff, and Sprague, 1991), the im¬ 
provement of GTAs’ teaching, and the feedback given to GTAs (Bond-Robinson, 2000; Black and 
Kaplan, 1997; Prieto and Meyers, 2001; Prieto and Meyers, 1999) have been studied extensively. In 
a study across three departments (two in chemistry and one in astronomy), Seymour et al. (2005) 
explored GTAs’ roles in both traditional course environments and in environments implementing in¬ 
novative pedagogical practices. Within traditional course environments, GTAs reported engaging in 
five primary activities to: (1) working in small groups with undergraduate students, (2) working with 
undergraduate students in laboratories, (3) providing academic support for students, particularly 
during office hours, (4) grading and preparing homework and exams, and (5) serving as a second 
authority figure in undergraduate courses. Seymour et al. (2005) found that within innovative course 
environments, GTAs engaged in three roles. The first was that of creative troubleshooter, meaning 
that GTAs resolved daily problems and alerted faculty to challenges that were out of their control. 
Second, GTAs acted as consultants who offered faculty information about the classroom interac¬ 
tions and occurrences within the environments in which they work. Finally, GTAs became collegial 
collaborators, meaning that they worked closely with faculty to facilitate student learning. However, 
while GTAs fulfill their roles through an extensive array of activities, the training they receive before 
entering the classroom or laboratory is limited and by no means consistent across departments. 
Because of the important roles that GTAs play in both traditional and innovative environments, 
feedback to GTAs about the extent to which they are fulfilling their teaching roles is needed. 

To date, most feedback for GTAs has come informally from course coordinators, supervisors, or 
peers (Prieto and Meyers, 2001), from videotapes of course lessons, self-reflections by the GTAs 
(Prieto and Meyers, 1999; Bond-Robinson, 2000), or undergraduate student evaluations completed 
at the end of the term (Black and Kaplan, 1997; Seymour et al., 2005). Student reflections, which 
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are used most often to give feedback to GTAs about their teaching practices, might occur via 
group discussions, written course evaluations, or surveys (Black and Kaplan, 1997). Student sur¬ 
veys composed of Likert scale items are the most commonly used method for providing feedback 
GTAs. The items in these surveys may ask about a variety of issues including GTAs’ preparedness, 
communication skills, approachability, or helpfulness (Black and Kaplan, 1997). When comparing 
undergraduate students’ ratings and graduate students’ self-reported ratings of teaching effective¬ 
ness, Twale, Shannon, and Moore (1997) found GTAs’ self-reported ratings to be higher than their 
students’ ratings. Self-ratings by international GTAs were also higher than self-ratings by U.S. GTAs, 
although undergraduate students gave U.S. GTAs higher ratings than international GTAs. 


CONCEPTUAL FRAMEWORK 


The conceptual framework on which this study is based is the “How People Learn” (HPL) Frame¬ 
work (Bransford, Brown, and Cocking, 1999). The HPL framework identifies four dimensions which 
are essential elements of an effective learning environment: (1) learner-centeredness, (2) knowledge- 
centeredness, (3) assessment-centeredness, and (4) community-centeredness. Within the learner- 
centered dimension, instructors recognize and attend to unique backgrounds and experiences that 
students bring to the classroom. The knowledge-centered dimension includes well-articulated les¬ 
sons and emphasizes both students’ understanding of course content and their abilities to transfer 
and apply this knowledge accurately in new environments. Two key aspects of the assessment- 
centered dimension are opportunities for feedback and revision by both instructors and students. 
The community-centered dimension emphasizes the creation of positive group dynamics among all 
members of the classroom community and encourages students to learn from one another. The HPL 
framework has been demonstrated to be a useful tool for measuring levels of these four dimensions 
in both traditional, lecture-based, and innovative lesson designs (VaNTH, 2010) with innovative les¬ 
sons specifically incorporating all four dimensions (Greenberg, Smith and Newman, 2003; Pandy et 
al., 2004). For this reason, the authors have used the HPL framework to evaluate GTAs’ instruction 
across multiple engineering disciplines in engineering laboratory courses. 


PREVIOUS DATA COLLECTION 


To determine what type of tool is needed to record GTAs’ pedagogical experiences in engineer¬ 
ing laboratories, pedagogical data were collected in engineering laboratories using the VaNTH 


4 


WINTER 2011 





ADVANCES IN ENGINEERING EDUCATION 

Enhancing the Quality of Engineering Graduate Teaching Assistants 
through Multidimensional Feedback 



Observation System (VOS) and via focus groups, one composed of GTAs and another composed 
of the GTAs’ supervisors. Overviews of the findings of these activities are provided in the follow¬ 
ing sections. 

Direct Observations Using the VaNTH Observation System (VOS) 

Prior to developing a comprehensive GTA feedback tool, the authors used the VaNTH Observation 
System (VOS) (Harris and Cox, 2003) to collect real-time, in-class data about GTAs’ pedagogical 
practices in first-year engineering laboratory courses at Purdue University during the Fall 2007 and 
Fall 2008 semesters. This course serves as an introductory engineering course. Annually, the approxi¬ 
mate enrollment is about 1600 students with supervision by fifteen GTAs. The course meets four 
hours per week, which includes two hours of lecture and two hours of laboratory. In this course, both 
lecture and laboratory components are utilized so that students can engage in teamwork and can 
hone their problem-solving skills. The Classroom Interaction Observation (CIO) portion of the VOS 
was used to collect data using who, to whom, what, how, and media categories of the instrument, 
with data recorded in four- to six- second intervals. Codes for each category are described in detail 
in Harris and Cox (2003). The who and to-whom categories note who is initiating or responding to 
in-class interactions. The what category describes twelve types of in-class interactions that may 
occur during the four- to six- second observation cycle. The content of the how category within the 
VOS differs from the CIO portions of other similar observational instruments in its identification of 
the presence of “How People Learn” dimensions in the learning environment. To understand ad¬ 
ditional information about the types of media that an instructor is using during an observed class 
session, the VOS contains a media category. 

The VOS was used to understand what data it was capable of capturing in laboratory course 
environments and how this data differed from that collected in conventional classroom environ¬ 
ments for which the VOS was designed. To collect data using the CIO portion of the VOS, observers 
sat through three-hour lab course sessions and observed a GTA supervising thirty undergraduate 
engineering students. The observers recorded approximately 20 to 25 five-minute segments of 
instruction at regular intervals by selecting corresponding codes under each category, thereby 
resulting in the creation of hundreds of code strings per course session. 

The use of the VOS in laboratory course settings was found to be hampered by several factors 
(McNeill, et al, 2008). First, the VOS required considerable training to use. Extensive training was 
required in order to achieve 85% inter-rater reliability accuracy in four- to six- second coding intervals. 
Second, not all categories were intuitive. While the who, to whom, and media categories were straight¬ 
forward, the what and how categories required additional interpretation by observers, depending 
on the types of classroom interactions that were occurring. Also, the how category, containing the 
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four HPL dimensions, was difficult to interpret because of the subjective interpretation of observed 
interactions representing these dimensions. Finally, observational data were collected primarily on 
handheld computers running software which required users to navigate multiple interfaces. 

Undergraduate Student Survey 

To complement observations recorded with the VOS, survey data were collected in first-year 
engineering laboratory courses taught by both U.S. and international GTAs at Purdue University 
(Cox et al., 2010b). Seventy-eight undergraduate engineering students shared their perceptions of 
their GTAs using a modified Adjective Generation Technique (Potkay and Allen, 1988). More specifi¬ 
cally, students were asked to describe their GTAs using five words. Based upon student feedback, 
researchers identified three themes of highest concern to undergraduate students about their GTAs: 
(1) GTAs’ knowledge levels, (2) their overall effectiveness, and (3) their approachability. 

Focus Group Interviews about the Roles and Experiences of GTAs and Supervisors 

During summer 2008 at Purdue University, two focus groups were conducted to assist with 
the understanding of the roles of GTAs, GTAs' relationship with their supervisors (the person who 
provided feedback most often), and the feedback needs of both groups. One of the focus groups 
was comprised of all GTAs and the other was of the GTAs’ supervisors. A focus group study is a 
carefully planned series of discussions designed to obtain perceptions on a defined area of interest 
in a permissive, nonthreatening environment. Each group is conducted with 5 to 10 people led by 
a skilled interviewer. The discussions are relaxed and often participants enjoy “sharing their ideas” 
(Krueger and Casey; 2008; p. 2). Focus groups are commonly used and are an effective method 
when the researcher wants to obtain varied perceptions on a topic (Morgan, 1996). The participants 
in the supervisors’ focus group represented the disciplines of electrical and computer engineer¬ 
ing, chemical engineering, nuclear engineering, and first-year engineering. Participants in the GTA 
focus group represented the disciplines of engineering education, materials engineering, electri¬ 
cal and computer engineering, and software engineering. Moderators asked the supervisors and 
the GTAs semi-structured questions about their roles and experiences that varied little between 
groups (Tables 1 and 2). 

Using open coding, the authors made the following conclusions: 

• The roles of supervisors and GTAs differed. The participants believed that a supervisor’s job 
was to think explicitly about the purpose of their courses, whereas, the participants thought 
that the role of the GTA is simply to implement the ideas of their supervisor. 

• Supervisors and GTAs acknowledged GTAs to be the liaisons or “conduits” between the un¬ 
dergraduate students and professors. 


6 


WINTER 2011 





ADVANCES IN ENGINEERING EDUCATION 

Enhancing the Quality of Engineering Graduate Teaching Assistants 
through Multidimensional Feedback 



Please introduce yourself by stating your name and the course(s) that you supervise. 
How would you describe the course that you supervise? 

What is the role of the TA in the course, or courses, that you supervise? 

What criteria do you use to select your TAs? 

What expectations do you have of your TAs? 

What do think your TAs enjoy about their work? 

What do you think your TAs dislike about their work? 

Do you have any formal training for your TAs? 

Do you conduct any performance evaluations of your TAs? 

How often do you interact with your TAs? 

Table 1: Questions asked of supervisors. 


Please introduce yourself by stating your name and the course that you teach. 
Please describe the course that you are a TA for. 

How would you describe your role in the course that you are a TA for? 

Do you enjoy your work? 

If you do, what do you enjoy about it? 

What do you dislike about your work? 

What are your strengths and weaknesses? 

What would you do to improve your teaching? 

What kind of relationship do you have with your students? 

Please describe the relationship that you had with your supervisor. 

What would you do to improve your interactions with your supervisor? 

If you all had any type of training what type of training did you have? 

Table 2: Questions asked of GTAs. 


• Answering questions and giving students feedback, guidance, and assistance were some of 
the most important roles of GTAs. 

• While GTAs were rarely provided with formal or explicit performance evaluations, many su¬ 
pervisors did provide their GTAs with informal feedback. 

• Training for GTAs occurred campus-wide, and departments rarely followed up with additional 
training representing the disciplinary content that would be taught by the GTAs. 

• Some GTAs relied on peer observations to learn about teaching. 
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CONNECTING RESEARCH TO PRACTICE 


A synthesis of the findings from the studies above suggests that the following elements be in¬ 
corporated into a tool that would capture the pedagogical practices of GTAs: 

• The tool should be easy to use and should not require extensive observer training. 

• A complete description of the codes and what they represent should be integrated into the 
tool so that users do not have to memorize observational codes and definitions. 

• The who and to whom categories from the VaNTH Observation System (VOS) could be cus¬ 
tomizable according to the environments (i.e., laboratory or classroom) where the tool is 
used. 

• A help function should be provided to assist users with the operation of the tool. 

• GTAs should obtain deliberate, periodic feedback about their pedagogical practices. 

• Because GTAs have to understand engineering concepts, the tool might explore the extent to 
which they are accurately and effectively conveying pedagogical concepts to their students. 

• The student observation piece should allow students to assess formatively their laboratory 
experiences. 

• A common code or identifier should be used to connect all elements of the tool. 


DEVELOPMENT OF THE GLOBAL REAL-TIME ASSESSMENT TOOL FOR 
TEACHING ENHANCEMENT (G-RATE) 

In response to the aforementioned needs, the G-RATE, an adaptation of the VaNTH Observation 
System, was developed to provide multidimensional (e.g., quantitative/qualitative, student/supervi¬ 
sor) feedback to GTAs about their instructional interactions in a classroom. The G-RATE has a flexible 
and easy-to-use interface that may be used in both lecture and laboratory course environments 
and may be used to collect data from a variety of stakeholders (including undergraduate students, 
GTAs, researchers, and administrators). The G-RATE is a piece of software that was designed using 
the Microsoft Visual C# programming language and runs on the Windows XP or Vista operating 
system and requires 3 MB of hard disk space and an eight inch or larger computer screen. Data 
produced by the G-RATE is stored using Microsoft Access, and Microsoft Excel is needed for gen¬ 
erating observation reports. 

The graphical user interface of the G-RATE has five working environments, or functions, which 
correspond to the Observer, Student, Graduate Teaching Assistant, Researcher, and Administrator. 
Training materials can be accessed on the main page of the user interface where the functions are 
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Function 

Description 

Administrator 

This function allows a lab coordinator to modify the data collection 
parameters of the G-RATE (e.g., adding the name of observed GTAs; 
tailoring self-reflective questions for the GTAs; selecting appropriate 
questions for undergraduate students). This occurs at the beginning of the 
lab session. 

Observer 

This function produces several time- or event-sampled codes that reflect 
observed instructional practices of GTAs during lab sessions. Codes 
represent elements of the "How People Learn” framework (Bransford, 
Brown, and Cocking, 1999) 

Student 

This function contains open-ended and closed-ended questions to be 
completed by undergraduate students enrolled in the observed lab session. 
These questions will be formative and summative and will be answered 
during the last ten minutes of the lab session. 

Graduate 

Teaching 

Assistant 

This function contains open-ended items about GTAs' pedagogical 
practices during lab sessions. The GTA will complete this reflection after 
the lab session. 

Researcher 

This function gives a researcher access to data from the other G-RATE 
functions without overriding modifications made by the lab coordination. 
This function houses the profiles given to GTAs and faculty about GTAs’ 
pedagogical practices. The researcher accesses these profiles after all 
other data have been collected. 


Table 3: G-RATE functions and descriptions. 


located. These materials provide tutorials about system operation as well as an overview of the 
“How People Learn” framework (Bransford, Brown, and Cocking, 1999), the conceptual framework 
on which the G-RATE is based. 

Table 3 provides descriptions of the five functions of the G-RATE and Figure 1 displays the ob¬ 
servational cycle suggested for the use of the G-RATE. More detailed descriptions of each function 
are described in subsequent parts of this paper. 

Administrator {Admin) Function Page 

The Admin function page is intended for the use of GTA supervisors who will use the instrument 
to provide feedback from the courses that they supervise. In the Admin function, a supervisor is 
able to modify some cosntents of the Observer, Student, and GTA functions. An administrator ac¬ 
count must be created in order for the G-RATE to be used. A private database, which will store data 
collected from the Observer, Student, and GTA functions, is automatically generated when creating 
the administrator account. 

The Admin function page contains four tabs- Observer, GTA, Student, and User page (Figure 2). 
On the Observer tab the administrator can change the who and to whom code string categories 
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Figure 7: Observational cycle for the G-RATE. 



Figure 2: Admin function page of the G-RATE. 


of the Observer function. Similarly, the questionnaires for the GTA and Student functions can be 
altered on the GTA and Student tabs, respectively. All of these modifications are stored in the pri¬ 
vate database. 

Since security and anonymity are important concerns when collecting data using the G-RATE, 
a user has two possible data store options- a public database or a private database (Figure 3). A 
public database saves the data in a default file that can be read in the Researcher function and stored 
in the Observer, Student, and GTA functions. Only authorized administrators, observers, students, 
GTAs, and researchers can read and store data in a private database. In order to grant access to 
a private database, security options can be applied by accessing the User page tab in the Admin 
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Figure 3: Public and private database options in the G-RATE. 


function. By setting up the passwords, the administrator can control (1) who may use the contents 
of the private database, (2) who may store data in the private database, and (3) who may read data 
in the private database. 

Observer Function Page 

The Observer function page is used by observers to record observational data of GTAs’ instruc¬ 
tional practices in laboratory course settings. Figure 4 shows a screen shot of the Observer function 
page which is a modified version of the Classroom Interaction Observation (CIO) portion of the 
VaNTH Observation System (VOS) instrument (Harris and Cox, 2003). 

Rather than using abbreviated codes as the VOS does, the G-RATE’s Observer function displays 
entire descriptive codes (e.g., “instruction” or “guidance”). To help observers learn and recall codes 
quickly, tooltip functions are located beside each code, and Help functions are provided in the upper 
corner of the page to remind observers of the definitions of the “How People Learn” dimensions 
and other observational elements. 

There are two code string saving options in the Observer function. Users may choose whether 
they want to save observation code strings manually or automatically. If saving codes manually, an 
observer must select one code each representing the who , to whom and what categories and at 
least one code representing the how category and then press the “submit” button to save the data. 
If a category is not represented, a warning message will appear in a pop-up window informing the 
user of the unselected categories. If automatically saving codes, the observer can select a record¬ 
ing time interval that will save codes at regular time intervals regardless of categories selected or 
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Figure 4: Observer function page of the G-RATE (Cox et al., 2010a). 


other parameters. Within the database, any category without selected codes will display the word 
‘miss’ under the selected category. 

In addition to collecting code string data, observers may write notes at anytime during an ob¬ 
servation session. Notes are saved with corresponding code strings regardless of whether data is 
being stored manually or automatically. 


Student and GTA Function Pages 

Both the Student and GTA function pages contain enumerated questions which may be completed 
by students and GTAs, respectively. The Student function contains open-ended and Likert scale 
items and the format of this function is similar to a regular classroom evaluation form. A comment 
section is provided for each Likert item. The GTA function contains open-ended self-reflective items 
on GTAs' pedagogical practices. 


Researcher Function Page 

The primary role of the Researcher function page is to display data from both the public and 
private databases for use by a researcher. As stated in the previous section, any data produced 
by other functions of G-RATE is stored as a Microsoft Access database file. Thus, there may be a 
chance that the file may be deleted, that the data may be modified by accident, or that other people 
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who are not authorized may open and read the data. For these reasons, all databases are created 
as a password protected file by default, and the file cannot be opened directly. In other words, the 
database can only be read using the Researcher function. 

The Researcher function page also contains data filtering options. There could be hundreds or 
even thousands of code strings of data if observations are performed for a long period of time or if 
multiple GTAs are observed under a group database. In this case, a researcher may have difficulty 
finding necessary information among the enormous amounts of data. Using a filter, a user can ex¬ 
tract data from any function using multiple options (e.g., key words: observer names, GTA names, 
or dates). The researcher can then export this data as an Excel file, and may copy the database file 
to load it on another computer. 

An Example 

As an example, demonstration of data generated by the Observer function is located in 
Figure 5. At different times during the class, the observer submitted observational data in real time. 
The observer chose different codes under the who, to whom, and what categories. Meanwhile, the 



Figure 5: Sample screenshot of observer data. 
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observer also documented the dimension of the HPL framework at that particular time point and 
wrote notes in the note box. The data are presented in a pie-chart that summarizes the overall 
distribution of the teaching practices in the context of the HPL framework. This information can 
be used later as feedback to the instructor or as part of data sources for further research about 
effective pedagogical practices. 


DISCUSSION AND IMPLICATIONS 


Although many GTAs may receive feedback about their teaching in semester-long courses via 
several activities (e.g., journaling and evaluation of critical incidents via video) (Chavela et al., 2008; 
McClure, 2007), such courses are not offered at every university, and faculty are not always available 
to give GTAs pedagogical feedback that is grounded in theory and effective practices. Training for 
GTAs is often inconsistent (Prieto et al., 2001). For this reason, a tool such as the G-RATE is needed. 
Unlike instruments which may provide graduate students with feedback once during the semes¬ 
ter, if at all, the G-RATE provides GTAs with formative and summative feedback from supervisors, 
researchers, and from their students. In addition, rather than obtaining this feedback via primarily 
quantitative methods (e.g., end-of-semester surveys), graduate students observed using the G- 
RATE will obtain quantitative and qualitative feedback that engages them in effective pedagogical 
practices, particularly practices related to the dimensions of Bransford, Brown, and Cooking’s (1999) 
“How People Learn” framework. 

Within engineering laboratory environments, metrics of interest include student retention, stu¬ 
dent satisfaction, and the efficacy of laboratory simulations (Feisel and Rosa, 2005). Researchers 
anticipate that the G-RATE can be used to explore these and other metrics related to student out¬ 
comes. Furthermore, the tool can be used to create and to extend research studies about relation¬ 
ships between GTAs and their supervisors or relationships between GTAs and their undergraduate 
students (Cox et al., 2010a; Commander, Hart, and Singer, 2000). One may also explore the impact 
of G-RATE feedback upon graduate students’ professional development experiences in engineering 
or even use it to identify the most effective ways to give beneficial feedback to GTAs about their 
teaching. A focus on graduate students’ professional development has strong implications for the 
creation of formal, standardized training for engineering GTAs regardless of discipline. Furthermore, 
the implications can also assist with the development of workshops and/or other resources for 
supervisors who are expected to provide formative and summative feedback to engineering GTAs, 
and data collected via the G-RATE may be used to explore ways that the tool might be revised to 
enhance the feedback given to GTAs. 
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FUTURE WORK 


Creation of Pedagogical Profiles from G-RATE Data 

Once data have been collected, graduate students’ professional development may be enhanced 
via the generation of comprehensive profiles of GTA instruction. Sample profiles from the Observer 
function will include graphical representations of classroom occurrences and events. More specifi¬ 
cally, at the end of an observation, each GTA will receive an individual report and analysis, graphs 
representing observations, survey output from undergraduate students about their GTAs (e.g., general 
feelings about GTAs’ teaching, ideas of how GTAs are doing, and perceptions of GTAs’ disciplinary 
content knowledge), and a summary of pedagogical information (framed within the context of the 
HPL framework) over the course of the semester. 

Large-Scale Distribution of the G-RATE 

The G-RATE research team has received feedback about the tool via an external consultant and 
from engineering audiences at two public universities housing several engineering departments. 
Members of these audiences offered various suggestions. Among these include (1) the creation 
of an awareness brochure to inform audiences about the G-RATE and its potential to enhance the 
feedback experiences of GTAs in engineering, (2) a collaboration between the research team and 
universities that support engineering graduate students within the National Science Foundation’s GK- 
12 program, (3) translation of the Student and Graduate Teaching Assistant functions of the G-RATE 
to virtual learning environment systems (e.g., Blackboard) that may be used currently in laboratory 
environments so that students do not have to become familiar with new systems and interfaces, 
and (4) the modification of the G-RATE for new faculty, K-12 in-service teachers, and pre-service 
teachers during student teaching. Suggestions related to the practical use of the instrument include 
limiting the number of student response items to ten so that students do not feel overwhelmed by 
the assessment and providing five to eight sentence stems for GTAs to complete at the end of the 
lab to prompt their thinking (e.g., The most difficult aspects of today’s lab was_). 

Testing of the G-RATE is occurring in several ways. As the final version of the G-RATE is being 
developed, the team is collecting data via IRB-approved video sessions of bioengineering class¬ 
rooms and of electrical and computer engineering laboratories. The team plans to pilot the finished 
tool among engineering students in fall 2010. From here, revisions will be made, and the final tool 
will be distributed for use in spring 2011. As part of a larger evaluation effort, data from students 
across multiple institutions will be collected to confirm the reliability and validity among targeted 
populations. 
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CONCLUSIONS 


The G-RATE is a new tool that is informed from observations and empirical studies exploring the 
needs of engineering GTAs and that minimizes the limitations of current professional development 
practices for GTAs. Comprised of five functions that collect multidimensional feedback from labora¬ 
tory observers, the G-RATE provides feedback about GTAs’ pedagogical practices to not only the 
GTA, but to supervisors and to researchers who are interested in the long-term professional devel¬ 
opment of these graduate students. Such development is needed, since most GTAs do not receive 
consistent feedback about their teaching. Future implications of this work include the use of G-RATE 
data in conjunction with student learning outcomes within the laboratory and the translation of the 
G-RATE to multiple audiences that might benefit from multidimensional feedback. 
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